Systems and methods for automated assessment of embryo quality using image based features

ABSTRACT

Systems and methods for automated imaging and evaluation of image based features are disclosed herein. Method for automated imaging and evaluation of image based features can include receiving time-lapse images of at least one human embryo contained in a multi-well culture dish that can have a plurality of micro-wells. Image based features can be automatically generated from the time-lapse images of the human embryo. The image based features, which can include a cavitation feature, can be inputted into a classifier. The classifier can automatically and directly generate a viability prediction with the classifier from the image-based features.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/221,530, filed on Sep. 21, 2015, and entitled “SYSTEMS AND METHODSFOR AUTOMATED ASSESSMENT OF EMBRYO QUALITY USING IMAGE BASED FEATURES”,the entirety of which is hereby incorporated by reference herein.

BACKGROUND

Infertility is a common health problem that affects 10-15% of couples ofreproductive-age. In the United States alone in the year 2006,approximately 140,000 cycles of in vitro fertilization (IVF) wereperformed (cdc.gov/art). This resulted in the culture of more than amillion embryos annually with variable, and often ill-defined, potentialfor implantation and development to term. The live birth rate, percycle, following IVF was just 29%, while on average 30% of live birthsresulted in multiple gestations (cdc.gov/art). Multiple gestations havewell-documented adverse outcomes for both the mother and fetuses, suchas miscarriage, pre-term birth, and low birth rate. Potential causes forfailure of IVF are diverse; however, since the introduction of IVF in1978, one of the major challenges has been to identify the embryos thatare most suitable for transfer and most likely to result in termpregnancy.

Traditionally in IVF clinics, human embryo viability has been assessedby simple morphologic observations such as the presence ofuniformly-sized, mononucleate blastomeres and the degree of cellularfragmentation (Rijinders P M, Jansen C A M. (1998) Hum Reprod13:2869-73; Milki A A, et al. (2002) Fertil Steril 77:1191-5). Morerecently, additional methods such as extended culture of embryos (to theblastocyst stage at day 5) and analysis of chromosomal status viapreimplantation genetic diagnosis (PGD) have also been used to assessembryo quality (Milki A, et al. (2000) Fertil Steril 73:126-9; FragouliE, (2009) Fertil Steril June 21 [EPub ahead of print]; El-Toukhy T, etal. (2009) Hum Reprod 6:20; Vanneste E, et al. (2009) Nat Med15:577-83). However, potential risks of these methods also exist in thatthey prolong the culture period and disrupt embryo integrity(Manipalviratn S, et al. (2009) Fertil Steril 91:305-15; Mastenbroek S,et al. (2007) N Engl J Med. 357:9-17).

Recently it has been shown that time-lapse imaging can be a useful toolto observe early embryo development. Some methods have used time-lapseimaging to monitor human embryo development following intracytoplasmicsperm injection (ICSI) (Nagy et al. (1994) Human Reproduction.9(9):1743-1748; Payne et al. (1997) Human Reproduction. 12:532-541).Polar body extrusion and pro-nuclear formation were analyzed andcorrelated with good morphology on day 3. However, no parameters werecorrelated with blastocyst formation or pregnancy outcomes. Othermethods have looked at the onset of first cleavage as an indicator topredict the viability of human embryos (Fenwick, et al. (2002) HumanReproduction, 17:407-412; Lundin, et al. (2001) Human Reproduction16:2652-2657). However, these methods do not recognize the importance ofthe duration of cytokinesis or time intervals between early divisions.

Other methods have used time-lapse imaging to measure the timing andextent of cell divisions during early embryo development (WO2007/144001). However, these methods disclose only a basic and generalmethod for time-lapse imaging of bovine embryos, which are substantiallydifferent from human embryos in terms of developmental potential,morphological behavior, molecular and epigenetic programs, and timingand parameters surrounding transfer. For example, bovine embryos takesubstantially longer to implant compared to human embryos (30 days and 9days, respectively). (Taft, (2008) Theriogenology 69(1):10-16. Moreover,no specific imaging parameters or time intervals are disclosed thatmight be predictive of human embryo viability.

More recently, time-lapse imaging has been used to observe human embryodevelopment during the first 24 hours following fertilization (Lemmen etal. (2008) Reproductive BioMedicine Online 17(3):385-391). The synchronyof nuclei after the first division was found to correlate with pregnancyoutcomes. However, this work concluded that early first cleavage was notan important predictive parameter, which contradicts previous studies(Fenwick, et al. (2002) Human Reproduction 17:407-412; Lundin, et al.(2001) Human Reproduction 16:2652-2657).

Finally, no studies have validated the imaging parameters throughcorrelation with the molecular programs or chromosomal composition ofthe embryos. Methods of human embryo evaluation are thus lacking inseveral respects, including their inability to conduct the imaging andevaluation in an automated fashion.

It is against this background that a need arose to develop theapparatus, method, and system for the improved viability prediction ofembryos, oocytes, and stem cells described herein.

BRIEF SUMMARY

One aspect of the present disclosure relates to a method for determiningviability of human embryos with an imaging system. The method includesreceiving time-lapse images of at least one human embryo contained in amulti-well culture dish including a plurality of micro-wells,automatically generating image-based features from the time-lapse imagesof the human embryo, inputting the image-based features into aclassifier, and automatically and directly generating a viabilityprediction with the classifier from the image-based features.

In some embodiments, directly generating the viability predictionincludes associating the human embryo imaged in the received time-lapseimages with one of a plurality of ranked categories, which categoriesare ranked according to a likelihood of viability. In some embodiments,the plurality of ranked categories includes up to 5 categories, and insome embodiments, the plurality of ranked categories includes at least 5categories. In some embodiments, the viability prediction includes aprediction of euploidy in the human embryo, and in some embodiments, theviability prediction includes a prediction of aneuploidy in the humanembryo.

In some embodiments the method includes generating a recommendation fromthe viability prediction, which recommendation includes one of: arecommendation to select the human embryo; and a recommendation todeselect the human embryo. In some embodiments, the classifier includesone of: a Random Forest classifier; an AdaBoost classifier; a NaïveBayes classifier; Boosting Tree, and a Support Vector Machine. In someembodiments, the image-based features relate to at least one of:cavitation; hatching; embryo expansion; and embryo collapse. In someembodiments, the image-based features relate to at least one of: an areaof the embryo; an area of a cavity of the embryo; a perimeter of theembryo; and a convex hull.

In some embodiments, the image-based features relate to at least one of:average cepstrum of an embryo attribute; final embryo area; maximumembryo area; average embryo area; number of cavitation peaks; finalcavitation area; maximum cavitation area; maximum embryo areadifference; maximum cavitation area difference; mean ration ofcavitation and embryo areas; mean area of cavitation; and maximum ratioof cavitation and embryo areas. In some embodiments, the image-basedfeatures relate to at least one of: a second highest embryo area; athird highest embryo area; an average cepstrum of the number of convexhull points; a second highest cavitation area; a third highestcavitation area; an average cepstrum of traversal cost; an averagecepstrum of standard deviation of Hessian features; an average cepstrumof standard deviation of continuity; and an average cepstrum of meancontinuity.

In some embodiments, the viability prediction is automatically anddirectly generated with the classifier from the image-based featureswithout any manual biological measurements. In some embodiments, theviability prediction is automatically and directly generated with theclassifier only from the image-based features. In some embodiments, theviability prediction is automatically and directly generated with theclassifier from the image-based features and from manual biologicalmeasurements.

One aspect of the present disclosure relates to a method for determiningviability of human embryos with an imaging system. The method includesreceiving a series of time-lapse images of at least one human embryocontained in a multi-well culture dish including a plurality ofmicro-wells, extracting data from the series of time-lapse images, whichdata identifies one or several attributes of the images, calculating acavitation feature from the extracted data, and generating a viabilityprediction with the calculated cavitation feature.

In some embodiments, the cavitation feature relates to at least one of:embryo image area; cavity image area; a change in embryo image area overtime; a change in cavity image area over time; embryo image perimeter;and convex hull. In some embodiments, the convex hull identifies aminimum number of points defining lines that together circumscribe theimage of the embryo. In some embodiments, the cavitation feature relatesto at least one of: average cepstrum of an embryo attribute; finalembryo area; maximum embryo area; average embryo area; number ofcavitation peaks; final cavitation area; maximum cavitation area;maximum embryo area difference; maximum cavitation area difference; meanration of cavitation and embryo areas; mean area of cavitation; andmaximum ratio of cavitation and embryo areas.

In some embodiments, the cavitation feature is calculated using at leastone of: a Fourier transform; a cepstrum transform; and a wavelettransform. In some embodiments, the Fourier transform includes aDiscrete Fourier Transform (DFT). In some embodiments, the cepstrumtransform is calculated as the inverse DFT of the log magnitude of theDFT of the extracted data. In some embodiments, the wavelet transformincludes a discrete wavelet transform, and in some embodiments, thediscrete wavelet transform includes one of: Haar wavelet; Daubechieswavelets; Symlets wavelets; Battle-Lemarie wavelets; and Biorthogonalwavelets. In some embodiments, generating the viability predictionincludes inputting the cavitation feature into a classifier.

One aspect of the present disclosure relates to a method for determiningviability of human embryos with an imaging system. The method includesreceiving a series of time-lapse images of at least one human embryocontained in a multi-well culture dish including a plurality ofmicro-wells, extracting data from the series of time-lapse images, whichdata identifies one or several attributes of the images, calculating acepstrum from the extracted data, and generating a viability predictionwith the calculated cepstrum.

In some embodiments, the data is extracted from a portion of the seriesof time-lapse images. In some embodiments, the data is extracted fromthe entirety of the series of time lapse images. In some embodiments,the cepstrum is calculated for at least one of: embryo image area;cavity image area; a change in embryo image area over time; a change incavity image area over time; embryo image perimeter; and convex hull. Insome embodiments, the convex hull identifies the minimum number ofpoints defining lines that together circumscribe the image of theembryo.

In some embodiments, the cepstrum is calculated as the inverse DFT ofthe log magnitude of the DFT of the extracted data. In some embodiments,generating a viability prediction includes inputting the cavitationcepstrum into a classifier.

One aspect of the present disclosure relates to an automated imagingsystem for evaluation of human embryos to determine a developmentpotential. The system includes a stage for receiving a multi-wellculture dish having a plurality of micro-wells containing a sampleincluding at least one human embryo, a time-lapse microscope foracquiring time-lapse images of the multi-well culture dish on the stage,which time-lapse microscope can, or contains software code to control itto: automatically generate image-based features from the time-lapseimages of the human embryo, input the image-based features into aclassifier, and automatically and directly generate a viabilityprediction with the classifier from the image-based features.

In some embodiments, directly generating the viability predictionincludes associating the human embryo imaged in the received time-lapseimages with one of a plurality of ranked categories, which categoriesare ranked according to a likelihood of viability. In some embodiments,the system further includes a display and a Graphical User Interface(GUI).

In some embodiments, the time-lapse microscope is controllable togenerate a recommendation from the viability prediction, and the GUI canprovide the recommendation with the display. In some embodiments, therecommendation includes one of: a recommendation to select the humanembryo; and a recommendation to deselect the human embryo. In someembodiments, the plurality of ranked categories includes 5 categories.In some embodiments, the viability prediction includes a prediction ofeuploidy in the human embryo.

In some embodiments, the viability prediction includes a prediction ofaneuploidy in the human embryo. In some embodiments, the image-basedfeatures relate to at least one of: cavitation; hatching; embryoexpansion; and embryo collapse. In some embodiments, the image-basedfeatures relate to at least one of: an area of the embryo; an area of acavity of the embryo; a perimeter of the embryo; and a convex hull.

One aspect of the present disclosure relates to a method for determiningviability of human embryos with an imaging system. The method includes:generating a first viability prediction for predicting the likelihood ofan embryo reaching blastocyst stage, which first viability prediction isgenerated at a first time after fertilization, and which first viabilityprediction is based on at least one of: the length of the time intervalbetween the first cytokinesis and the second cytokinesis; the length ofthe time interval between the second cytokinesis and the thirdcytokinesis; the age of the human source of the egg at the time of eggharvesting; a cell count at the first time; and a result of amorphological analysis. In some embodiments, the method includesgenerating a second viability prediction for predicting euploidy in theembryo, which second viability prediction is generated at a second timeafter fertilization, which second time occurs after the first time, andwhich second viability prediction is based on at least one cavitationfeature.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is best understood from the following detailed descriptionwhen read in conjunction with the accompanying drawings. It isemphasized that, according to common practice, the various features ofthe drawings are not to-scale. On the contrary, the dimensions of thevarious features are arbitrarily expanded or reduced for clarity.Included in the drawings are the following figures.

FIG. 1 illustrates a schematic diagram of an apparatus, according to anembodiment of the invention;

FIG. 2 illustrates a schematic diagram of an imaging system, accordingto an embodiment of the invention;

FIG. 3 illustrates a flow chart for operating an imaging system,according to an embodiment of the invention;

FIG. 4 illustrates a schematic diagram of a microscope placed inside animaging system, according to an embodiment of the invention;

FIGS. 5A-D illustrate schematic views of examples of darkfieldillumination systems that may be used by the microscope of FIG. 4,according to an embodiment of the invention;

FIG. 6 illustrates a schematic view of the microscope in FIG. 4 mountedinside the housing of the imaging system of FIG. 2, according to anembodiment of the invention;

FIG. 7 illustrates a schematic view of the microscope in FIG. 4 mountedinside the housing of the imaging system of FIG. 2, according to anembodiment of the invention;

FIG. 8 illustrates a schematic diagram of a loading platform in theimaging system of FIG. 2, according to an embodiment of the invention;

FIGS. 9A-F illustrate a schematic diagram of a multi-well culture dish,according to an embodiment of the invention;

FIG. 10 illustrates a system for automated imaging of human embryos,oocytes, or pluripotent cells including an apparatus for automated dishdetection and well occupancy determination, according to an embodimentof the invention;

FIG. 11 is a series of images depicting the development of an embryo;

FIG. 12 is a flowchart illustrating one embodiment of a process forpredicting viability of an embryo;

FIG. 13 is a schematic illustration of an exemplary process for thecreation of the mask is shown;

FIG. 14 is a schematic illustration of one embodiment of a generatedconvex hull;

FIG. 15 is graph indicating the euploid rate of embryos placed into eachof five categories; and

FIG. 16 is a bar graph indicating the distribution of embryos in thesample from which the bar graph of FIG. 15 was generated.

DETAILED DESCRIPTION

Before the present apparatuses, systems, and methods are described, itis to be understood that this invention is not limited to particularapparatus, system, or method described, as such may, of course, vary. Itis also to be understood that the terminology used herein is for thepurpose of describing particular embodiments only, and is not intendedto be limiting, since the scope of the present invention will be limitedonly by the appended claims.

Where a range of values is provided, it is understood that eachintervening value, to the tenth of the unit of the lower limit unlessthe context clearly dictates otherwise, between the upper and lowerlimits of that range is also specifically disclosed. Each smaller rangebetween any stated value or intervening value in a stated range and anyother stated or intervening value in that stated range is encompassedwithin the invention. The upper and lower limits of these smaller rangesmay independently be included or excluded in the range, and each rangewhere either, neither or both limits are included in the smaller rangesis also encompassed within the invention, subject to any specificallyexcluded limit in the stated range. Where the stated range includes oneor both of the limits, ranges excluding either or both of those includedlimits are also included in the invention.

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. Although any methods andmaterials similar or equivalent to those described herein can be used inthe practice or testing of the present invention, some potential andpreferred methods and materials are now described. All publicationsmentioned herein are incorporated herein by reference to disclose anddescribe the methods and/or materials in connection with which thepublications are cited. It is understood that the present disclosuresupersedes any disclosure of an incorporated publication to the extentthere is a contradiction.

It must be noted that as used herein and in the appended claims, thesingular forms “a”, “an”, and “the” include plural referents unless thecontext clearly dictates otherwise. Thus, for example, reference to “acomputer” includes a plurality of such computers known to those skilledin the art, and so forth.

Any publications discussed herein are provided solely for theirdisclosure prior to the filing date of the present application. Nothingherein is to be construed as an admission that the present invention isnot entitled to antedate such publication by virtue of prior invention.Further, the dates of publication provided may be different from theactual publication dates which may need to be independently confirmed.

Definitions

The terms “developmental potential” and “developmental competence” areused herein to refer to the ability or capacity of a healthy embryo orpluripotent cell to grow or develop.

The term “embryo” is used herein to refer both to the zygote that isformed when two haploid gametic cells, e.g., an unfertilized secondaryoocyte and a sperm cell, unite to form a diploid totipotent cell, e.g.,a fertilized ovum, and to the embryo that results from the immediatelysubsequent cell divisions, i.e. embryonic cleavage, up through themorula, i.e. 16-cell stage and the blastocyst stage (with differentiatedtrophoectoderm and inner cell mass).

The term “pluripotent cell” is used herein to mean any cell that has theability to differentiate into multiple types of cells in an organism.Examples of pluripotent cells include stem cells oocytes, and 1-cellembryos (i.e. zygotes).

The term “stem cell” is used herein to refer to a cell or a populationof cells which: (a) has the ability to self-renew, and (b) has thepotential to give rise to diverse differentiated cell types. Frequently,a stem cell has the potential to give rise to multiple lineages ofcells. As used herein, a stem cell may be a totipotent stem cell, e.g. afertilized oocyte, which gives rise to all of the embryonic andextraembryonic tissues of an organism; a pluripotent stem cell, e.g. anembryonic stem (ES) cell, embryonic germ (EG) cell, or an inducedpluripotent stem (iPS) cell, which gives rise to all of embryonictissues of an organism, i.e. endoderm, mesoderm, and ectoderm lineages;a multipotent stem cell, e.g. a mesenchymal stem cell, which gives riseto at least two of the embryonic tissues of an organism, i.e. at leasttwo of endoderm, mesoderm and ectoderm lineages, or it may be atissue-specific stem cell, which gives rise to multiple types ofdifferentiated cells of a particular tissue. Tissue-specific stem cellsinclude tissue-specific embryonic cells, which give rise to the cells ofa particular tissue, and somatic stem cells, which reside in adulttissues and can give rise to the cells of that tissue, e.g. neural stemcells, which give rise to all of the cells of the central nervoussystem, satellite cells, which give rise to skeletal muscle, andhematopoietic stem cells, which give rise to all of the cells of thehematopoietic system.

The term “oocyte” is used herein to refer to an unfertilized female germcell, or gamete. Oocytes of the subject application may be primaryoocytes, in which case they are positioned to go through or are goingthrough meiosis I, or secondary oocytes, in which case they arepositioned to go through or are going through meiosis II.

By “meiosis” it is meant the cell cycle events that result in theproduction of gametes. In the first meiotic cell cycle, or meiosis I, acell's chromosomes are duplicated and partitioned into two daughtercells. These daughter cells then divide in a second meiotic cell cycle,or meiosis II, that is not accompanied by DNA synthesis, resulting ingametes with a haploid number of chromosomes.

By a “mitotic cell cycle”, it is meant the events in a cell that resultin the duplication of a cell's chromosomes and the division of thosechromosomes and a cell's cytoplasmic matter into two daughter cells. Themitotic cell cycle is divided into two phases: interphase and mitosis.In interphase, the cell grows and replicates its DNA. In mitosis, thecell initiates and completes cell division, first partitioning itsnuclear material, and then dividing its cytoplasmic material and itspartitioned nuclear material (cytokinesis) into two separate cells.

By a “first mitotic cell cycle” or “cell cycle 1” it is meant the timeinterval from fertilization to the completion of the first cytokinesisevent, i.e. the division of the fertilized oocyte into two daughtercells. In instances in which oocytes are fertilized in vitro, the timeinterval between the injection of human chorionic gonadotropin (HCG)(usually administered prior to oocyte retrieval) to the completion ofthe first cytokinesis event may be used as a surrogate time interval.

By a “second mitotic cell cycle” or “cell cycle 2” it is meant thesecond cell cycle event observed in an embryo, the time interval betweenthe production of daughter cells from a fertilized oocyte by mitosis andthe production of a first set of granddaughter cells from one of thosedaughter cells (the “leading daughter cell”, or daughter cell A) bymitosis. Upon completion of cell cycle 2, the embryo consists of 3cells. In other words, cell cycle 2 can be visually identified as thetime between the embryo containing 2-cells and the embryo containing3-cells.

By a “third mitotic cell cycle” or “cell cycle 3” it is meant the thirdcell cycle event observed in an embryo, typically the time interval fromthe production of daughter cells from a fertilized oocyte by mitosis andthe production of a second set of granddaughter cells from the seconddaughter cell (the “lagging daughter cell” or daughter cell B) bymitosis. Upon completion of cell cycle 3, the embryo consists of 4cells. In other words, cell cycle 3 can be visually identified as thetime between the embryo containing 3-cells and the embryo containing4-cells.

By “first cleavage event”, it is meant the first division, i.e. thedivision of the oocyte into two daughter cells, i.e. cell cycle 1. Uponcompletion of the first cleavage event, the embryo consists of 2 cells.

By “second cleavage event”, it is meant the second set of divisions,i.e. the division of leading daughter cell into two granddaughter cellsand the division of the lagging daughter cell into two granddaughtercells. In other words, the second cleavage event consists of both cellcycle 2 and cell cycle 3. Upon completion of second cleavage, the embryoconsists of 4 cells.

By “third cleavage event”, it is meant the third set of divisions, i.e.the divisions of all of the granddaughter cells. Upon completion of thethird cleavage event, the embryo typically consists of 8 cells.

By “cytokinesis” or “cell division” it is meant that phase of mitosis inwhich a cell undergoes cell division. In other words, it is the stage ofmitosis in which a cell's partitioned nuclear material and itscytoplasmic material are divided to produce two daughter cells. Theperiod of cytokinesis is identifiable as the period, or window, of timebetween when a constriction of the cell membrane (a “cleavage furrow”)is first observed and the resolution of that constriction event, i.e.the generation of two daughter cells. The initiation of the cleavagefurrow may be visually identified as the point in which the curvature ofthe cell membrane changes from convex (rounded outward) to concave(curved inward with a dent or indentation). The onset of cell elongationmay also be used to mark the onset of cytokinesis, in which case theperiod of cytokinesis is defined as the period of time between the onsetof cell elongation and the resolution of the cell division.

By “first cytokinesis” or “cytokinesis 1” it is meant the first celldivision event after fertilization, i.e. the division of a fertilizedoocyte to produce two daughter cells. First cytokinesis usually occursabout one day after fertilization.

By “second cytokinesis” or “cytokinesis 2”, it is meant the second celldivision event observed in an embryo, i.e. the division of a daughtercell of the fertilized oocyte (the “leading daughter cell”, or daughterA) into a first set of two granddaughters.

By “third cytokinesis” or “cytokinesis 3”, it is meant the third celldivision event observed in an embryo, i.e. the division of the otherdaughter of the fertilized oocyte (the “lagging daughter cell”, ordaughter B) into a second set of two granddaughters.

The term “fiduciary marker” or “fiducial marker,” is an object used inthe field of view of an imaging system which appears in the imageproduced, for use as a point of reference or a measure. It may be eithersomething placed into or on the imaging subject, or a mark or set ofmarks in the reticle of an optical instrument.

The term “micro-well” refers to a container that is sized on a cellularscale, such as to provide for accommodating one or more eukaryoticcells.

DESCRIPTION OF DISCLOSED EMBODIMENTS

Referring to FIG. 1, a schematic diagram of an apparatus 100 accordingto an embodiment of the invention is described. The apparatus 100includes a standard incubator 105 with one or more shelves for holdingimaging systems 110-120, described in more detail hereinbelow. Theimaging systems 110-120 have loading platforms and are placed inside theincubator 105 to image one or more embryos cultured in dishes mounted ontheir loading platforms.

The imaging systems 110-120 can be coupled to a computer 125, which maybe mounted on or near the incubator 105. The computer 125 includessoftware for analyzing the images acquired by the imaging systems110-120. In one embodiment, the computer 125 includes software fordetermining the developmental potential and/or the presence ofchromosomal abnormalities in cultured embryos. The computer 125 iscoupled to one or more output devices that can include one or severaldisplays or touch-screen panels, e.g., touch-screen panels 130-140. Thetouch-screen panels 130-140 may be configured to enable users to controlthe operation of the imaging systems 110-120 with an easy-to-usegraphical user interface (“GUI”). In one embodiment, multiple imagingsystems, e.g., the systems 110-120, may be controlled from a singletouch-screen panel, and multiple touch-screen panels may be controlledfrom a single computer, e.g., the computer 125.

A schematic diagram of an imaging system 200 according to an embodimentof the invention is illustrated in FIG. 2. The imaging system 200includes a single-channel or multi-channel microscope system includingon-board electronics placed inside an outer housing 205. Referring toFIGS. 1 and 2, in one embodiment, the imaging system 200 may communicatewith the computer 125. Alternatively, the imaging system 200 maycommunicate with a controller outside of the incubator 105 (seedescription with reference to FIG. 32) and may include a reduced set ofon-board electronics. The remainder of the on-board electronics may beincluded in the controller. Housing 205 may be constructed ofnon-embryotoxic materials, such as aluminum and plastics. In oneembodiment, a loading platform 210, also referred to herein as thestage, extending outward from the housing 205 allows for a multi-wellculture dish 215 to be positioned for imaging by the microscope system.Alternatively, the multi-well culture dish 215 may be loaded in aculture chamber integrated in the housing 205 (see description withreference to FIG. 35). Embryos may be placed in dish 215 with pipette225. In one embodiment, the microscope system includes software tomonitor the loading of a dish 215 into loading platform 210 and make anyadjustments necessary for the proper imaging of the embryos cultured inthe dish.

It is appreciated that a single channel/microscope system may be used toimage embryos for a single patient. It is also appreciated that imagingsystem 200 may be built as a single-channel microscope system asillustrated in FIG. 2, or it may be built as an integrated multi-channelmicroscope system. Accordingly, to facilitate the monitoring of embryosinside the incubator, a LCD display 220 may be placed outside thehousing 205 for showing the patient name, ID number, and other patientinformation to help users identify which channel is assigned to eachpatient. Alternatively, a color code system or other identificationmechanism may also be used to identify patients.

FIG. 3 illustrates a flow chart for operating an imaging system,according to an embodiment of the invention. The imaging system may bethe imaging system 200 of FIG. 2, or other types of devices for imagingof embryos, oocytes, or pluripotent cells. A user loads a multi-welldish (such as the multi-well dish 215 of FIG. 2, the multi-well dish 900of FIG. 9, or the multi-well dish 930 of FIG. 9C) with one or moreembryos into loading platform 210 (300). Using a GUI on one of thetouch-screen panels 130-140, the user selects a microscope channel in animaging system to image the embryos (305). In doing so, the user inputspatient information (e.g., name, ID) in the GUI to facilitate patient'sidentification. The patient information can also be enteredautomatically using a bar-code scanner or other means. For example, aseparate device such as a hand-held scanner could be used a priori toscan the bar-code on a multi-well dish. Then, when the dish is loadedinto the imaging system 200, the bar-code can be scanned again (e.g.,via a scanner built in to the imaging system or its platform) toidentify the patient identification. The patient information can bedisplayed on an LCD screen on the imaging system, on the touch-screenpanel outside the incubator, and elsewhere.

The multi-well dish can be placed on the loading platform of theselected channel in a given position and orientation (310), which may beadjusted by a software in the selected channel to ensure proper imagingof the embryos in the multi-well dish (315). In one embodiment, thesoftware recognizes when the multi-well dish is loaded properly andalerts the user of its proper loading by a light emitting diode (LED)indicator or other alert mechanism. In addition, the dish may have akeying feature that allows loading of the dish in a single possibleposition and orientation.

After closing of the incubator door (320), the time-lapse imagingcapture of the embryos can be initialized by first performing auto-focusand auto-exposure and verifying the quality of the acquired images(325). In one embodiment, images may be acquired at every given intervalfor a number of days. For example, images may be acquired every 1, 2, 3,4, 5, 6, 7, 8, 9, 10, 15, 20, 25, or 30 minutes for 6 hours, 12 hours, 1day, 2 days, 3 days, 4 days, 5 days, 6 days, 1 week, 2 weeks or 3 weeks.

Lastly, software in the selected channel and/or the computer 125analyzes the captured images and measures predictive parameters toprovide a prediction of which embryos will reach blastocyst and/or aranking of embryo quality. The prediction can be provided to the uservia the GUI and the output device. The prediction enables the user todetermine which embryos have development potential for humanimplantation.

Referring now to FIG. 4, a schematic diagram of a microscope 400 placedinside an imaging system is described, according to an embodiment of theinvention. The microscope 400 may be used with the imaging system 200 ofFIG. 2, or other types of devices for imaging of embryos, oocytes, orpluripotent cells. The microscope 400 may be any computer-controlledmicroscope that is equipped for digital image storage and analysis. Inone embodiment, the microscope 400 includes an illumination sub-assembly405 and an imaging sub-assembly 410. In one embodiment, the illuminationsub-assembly 405 provides darkfield illumination and may include a redLED, a collimating lens, a diffuser, a darkfield aperture, a right-anglemirror, and a condenser lens, among other optical components.

Imaging sub-assembly 410 may include an imaging objective lens (10×), astage such as a translation stage to focus the objective lens, a motorcoupled to the translation stage to provide computer-controlled focus, aright-angle mirror, a 4× objective lens that acts as a high-quality tubelens, and a CMOS camera to capture images. It is appreciated that thefield of view is large enough to view a set of micro-wells. It is alsoappreciated that some embodiments may use a light having a color otherthan red, a CCD camera, and different field of view, depth of field,optical layout, magnification objectives (e.g., 20×, 40×, etc.), motor,a positioning mechanism for moving a group of micro-wells under thefield-of-view, and so on.

It is further appreciated that the microscope 400 may employ brightfieldillumination, oblique brightfield, darkfield illumination, phasecontrast, Hoffman modulation contrast, differential interferencecontrast, or fluorescence. In some embodiments, darkfield illuminationmay be used to provide enhanced image contrast for subsequent featureextraction and image analysis. Darkfield illumination can also beachieved using epi-illumination, where the illumination light comes upthrough the imaging objective and illuminates the sample from beneath,rather than from above.

FIGS. 5A-C illustrate schematic views of examples of darkfieldillumination systems that may be used by the microscope 400 of FIG. 4,according to an embodiment of the invention. Darkfield illuminationsystem 500 of FIG. 5A illustrates an example of a traditional darkfieldillumination approach for use with time-lapse microscopes such as themicroscope 400, darkfield illumination system 505 of FIG. 5B illustratesan example of an approach using epi-illumination, and darkfieldillumination system 530 of FIG. 5C illustrates another approach forepi-illuminated darkfield. In system 505, for example, a 45-degreemirror 510 with a circular hole in the middle can be placed under theimaging objective 515. A hollow cone of light is reflected off themirror and up towards the imaging objective 515, where it gets focusedto the sample 520. Light scattered by the sample 520 gets collected bythe same imaging objective 515 and passes through the hole in the mirror510 and towards a tube-lens and camera 525 for collecting the image. Inaddition, red or near-infrared light sources may be used to reducephototoxicity and improve the contrast ratio between cell membranes andthe inner portion of the cells. In other embodiments, images can becaptured using one or more illumination wavelengths and the variousimages can be combined or used to provide additional information.

In one embodiment, a darkfield aperture 502 illustrated in FIG. 5A maybe placed as shown. Alternatively, the darkfield aperture 502 may beplaced in other configurations, such as between the 45-degree mirror 504and the condenser lens 506, or after the condenser lens 506.

Images that are acquired by the microscope 400 may be stored either on acontinuous basis, as in live video, or on an intermittent basis, as intime lapse photography, where a subject is repeatedly imaged in a stillpicture. In one embodiment, the time interval between images is between1 to 30 minutes in order to capture significant morphological events asdescribed below. In an alternative embodiment, the time interval betweenimages can be varied depending on the amount of cell activity.

For example, during active periods images could be taken as often asevery few seconds or every minute, while during inactive periods imagescould be taken every 10 or 15 minutes or longer. Real-time imageanalysis on the captured images could be used to detect when and how tovary the time intervals. It is appreciated that the light intensity fora time-lapse imaging system may be significantly lower than the lightintensity typically used on an assisted reproduction microscope due tothe low-power of the LEDs (for example, using a 1 W red LED compared toa typical 100 W Halogen bulb) and high sensitivity of the camera sensor.Thus, the total amount of light energy received by an embryo using themicroscope 400 is comparable to or less than the amount of energyreceived during routine handling at an IVF clinic. For example, for 2days of imaging, with images captured every 5 minutes at 0.5 seconds oflight exposure per image, the total amount of low-level light exposurecan be equivalent to roughly 30 seconds of exposure under a typical IVFinverted microscope.

Following image acquisition, the images are extracted and analyzed fordifferent cellular parameters related to embryo, stem cell, and/oroocyte development, for example, cell size, thickness of the zonapellucida, degree of fragmentation, particle motion in the cytoplasm,symmetry of daughter cells resulting from a cell division, duration offirst cytokinesis, time interval between cytokinesis 1 and cytokinesis2, time interval between cytokinesis 2 and cytokinesis 3, and timeintervals and durations of the first and second polar body extrusions.

FIG. 5D illustrates a schematic view of bimodal illumination that may beused by the microscope 400 of FIG. 4, according to one embodiment of theinvention. In one embodiment, an illumination assembly 550 may include afirst light source 552, an aperture 558, a second light source 559, anda condenser lens 562, among other optical components. In one embodiment,the first light source 552 and the second light source 559 may be redLEDs. In one embodiment, the aperture 558 may be a darkfield aperturehaving a first surface 580 configured to block light and a secondsurface 582 opposite to the first surface 580. The aperture 558 maydefine at least one opening 592 through which the hollow cone of lightcan pass. The second light source 559 may be attached to the secondsurface 582 of the aperture 558.

In a first mode of the illumination assembly 550, the first light source552 generates light that traverses a collimating lens 554, the at leastone opening 592 in the aperture 558 and the condenser lens 562 prior toreaching the sample 520. The aperture 558 may be placed before or afterthe condenser lens 562. The light may also traverse a diffuser 556. Thelight that passes through the at least one opening 592 may be reflectedby the 45-degree mirror 560. In one embodiment, a hollow cone of lightpasses through the at least one opening 592 in the aperture 558, whilethe remainder of the light is blocked by the aperture 558. In the firstmode, the second light source 559 does not generate light. Lightscattered by the sample 520 then traverses the imaging objective 515 andthe tube-lens and camera 525 for collecting the image. As described, inthe first mode of the illumination assembly 550, the illuminationassembly 550 performs darkfield imaging.

In one embodiment, the aperture 558 illustrated in FIG. 5D may be placedas shown. Alternatively, the aperture 558 may be placed in otherconfigurations, such as between the 45-degree mirror 560 and thecondenser lens 562, or after the condenser lens 562.

In a second mode of the illumination assembly 550, the first lightsource 552 does not generate light. Instead, the second light source 559generates light that reaches the sample 520 without traversing the atleast one opening 592 in the aperture 558, such that light generated bythe second light source 559 is not blocked by the aperture 558. Asdescribed, in the second mode of the illumination assembly 550, theillumination assembly 550 performs brightfield imaging.

In one embodiment, the illumination assembly 550 is configured in thefirst mode to perform time-lapse darkfield imaging of at least one of ahuman embryo, an oocyte, or a pluripotent cell. After completion of thetime-lapse darkfield imaging, the illumination assembly can beconfigured in the second mode to perform brightfield imaging of the atleast one of a human embryo, an oocyte, or a pluripotent cell. Thebrightfield imaging may be for intermittent image capture to enablemorphological observation. For example, the illumination assembly 550may be configured in the first mode for at least two days (and possiblya third day), and then may be configured in the second mode sometimeduring the third day. In this way, darkfield imaging can be performed(in the first mode) of a human embryo for at least the first two daysafter fertilization to minimize exposure of the embryo to light. Asingle brightfield image may be captured (in the second mode) sometimeon the third day after fertilization. This brightfield image canfacilitate morphology-based grading of the human embryo by anembryologist. By including the aperture 558 and the attached lightsource 559 and controlling the light sources 552 and 559 in the firstmode and the second mode, the illumination assembly 550 supports bothdarkfield imaging and brightfield imaging in the same hardware assembly,without any mechanical moving parts. In addition, the brightfield imagefor grading by the embryologist can be obtained by the illuminationassembly 550 without moving a dish containing the embryo. This isadvantageous because the embryo may be sensitive to disturbances such asmovement.

In one embodiment, the illumination assembly 550 alternates betweenbeing configured in the first mode and in the second mode at least onceper hour. For example, the illumination assembly can take a darkfieldimage in the first mode, followed by a brightfield image in the secondmode. This can be repeated periodically, such as every 5 minutes, toobtain time-lapse movies of a human embryo in both darkfield andbrightfield modalities.

FIG. 6 illustrates a schematic view of the microscope 400 of FIG. 4mounted inside the housing 205 of the imaging system 200 of FIG. 2,according to an embodiment of the invention. The illumination andimaging sub-assemblies 405-410 are mounted to an aluminum (or othermaterial) chassis (i.e., part of housing 205) that holds everythingtogether. The chassis also mounts the loading platform 210 for the dish215.

Another schematic view of the microscope inside the housing 205 is shownin FIG. 7, according to an embodiment of the invention. In thisembodiment, at the back end of the microscope are the on-boardelectronics for controlling the motor, camera, LED, LCD display, and anyother parts such as indicator LEDs. Alternatively, as described withreference to FIG. 32, all or part of the on-board electronics forcontrolling the motor, camera, LED, LCD display, and any other partssuch as indicator LEDs may be included in a controller outside of thehousing 205.

Referring now to FIG. 8, a schematic diagram of a loading platformincluded in the imaging system 200 of FIG. 2 is described, according toan embodiment of the invention. The loading platform 800 may haveseveral associated features to help identify if the dish 805 is locatedand oriented properly, such as, for example:

a back-plate to help position the dish 805;

a recessed groove (less than a millimeter deep) that the dish 805 seatsinto;

a keying (mechanical) feature on the dish 805 that only allows loadingwith one possible orientation;

markers (such as cross-hairs) to help with orientation. The user canrotate the dish 805 to align the vertical bar on the dish 805 with thecentral line;

an indicator LED to help illuminate the vertical bar or other feature onthe dish 805;

fiducials on the dish, such as letters, numbers, dots, or lines that canbe identified using the microscope and software;

software that uses the microscope to capture images of the dish 805 andmonitor the loading procedure. An indicator LED could change colors toalert the user when the dish 805 is oriented correctly or incorrectly;and/or

software that can account for misalignments (and potentially allowloading with an arbitrary orientation) and adjust the image accordingly.

It is appreciated that other mechanical and electronic components may beincluded in loading platform 800 for securing dish 805 into place.

FIGS. 9A-B illustrate a schematic diagram of a multi-well culture dish900, according to an embodiment of the invention. The dish 900 may beused with the imaging apparatus 200 of FIG. 2 or other types of devicesfor imaging of embryos, oocytes, or pluripotent cells. The dish 900 mayinclude multiple rings 905. In one embodiment, the rings 905 may besubstantially circular. Alternatively, the rings 905 may be oblong. Oneof the rings 905A may substantially circumscribe one or more wells 910.The ring 905A may be substantially centrally disposed in the dish 900.The wells 910 may be micro-wells. In one embodiment, each micro-well 910can hold a single embryo, oocyte, or pluripotent cell, and the bottomsurface of each micro-well 910 can have an optical quality finish suchthat a group of embryos within a single group of micro-wells can beimaged simultaneously by a single miniature microscope with sufficientresolution to follow cellular events. Each micro-well 910 may also bedesigned with a depth to facilitate its use. In one embodiment, the dish900 may include one or more rings 905B. The rings 905B may be laterallyoffset from the ring 905A, and may be used to hold media drops forrinsing.

Referring to FIG. 9A, in one embodiment, an outer ring 915 may bepositioned around the rings 905. The marker 822 (described withreference to FIG. 8) may be disposed adjacent to a lateral surface 917of the outer ring 915.

Referring to FIG. 9B, in one embodiment, the micro-wells 910 may bedisposed in a grid 920, such as a rectangular grid or a square grid. Forexample, the grid 920 may be 3×4 (as shown in FIG. 9B), 3×3, or 4×5.However, the dimensions of the grid are not limited to these examples.

FIG. 9C illustrates a schematic diagram of a multi-well culture dish930, according to an embodiment of the invention. The dish 930 may beused with the imaging apparatus 200 of FIG. 2, or other types of devicesfor imaging of embryos, oocytes, or pluripotent cells. The dish 930 mayinclude a ring 932 that may be substantially centrally disposed in thedish 930. In one embodiment, the ring 932 may be substantially circular.Alternatively, the ring 932 may be oblong. The ring 932 maysubstantially circumscribe one or more wells 910 (described withreference to FIGS. 9A and 9B). The dish 930 may also include one or morerings 905B (described with reference to FIGS. 9A and 9B).

FIG. 9D illustrates a cross-section view of the multi-well culture dish930 along cross-section A-A in FIG. 9C, according to an embodiment ofthe invention. Referring to FIGS. 9C and 9D, the ring 932 is disposed ona lower surface 936 of the dish 930. The ring 932 defines a cavity 938,and has an upper surface 940, an outer lateral surface 942, and an innerlateral surface 944. The cavity 938 has a cavity bottom 946, and themicro-wells 910 are defined by the cavity bottom 946. The inner lateralsurface 944 of the ring 932 is disposed between the outer lateralsurface 942 and the micro-wells 910, and extends from the upper surface940 of the ring 932 to the cavity bottom 946.

In one embodiment, the inner lateral surface 944 slopes toward themicro-wells 910 such that a first width 950 of the ring 932 at the lowersurface 936 of the dish 930 is greater than a second width 952 of thering 932 at the upper surface 940 of the ring 932. In one embodiment,the first width 950 is in the range from about two times to about sixtimes as large as the second width 952, such as three times, four times,or five times as large. Alternatively, the inner lateral surface 944 maybe substantially vertical, such that the first width 950 isapproximately equal to the second width 952.

Movement of a media drop stored in the ring 932 may be caused bymovement of the dish 930, such as due to transport or other handling ofthe dish 930. Advantageously, this movement of the media drop can bereduced by the sloping of the inner lateral surface 944 toward tomicro-wells 910, which positions the inner lateral surface 944 closer tothe micro-wells 910. This reduces the area in which a media drop storedin the ring 932 can move, and provides a larger contact surface areabetween the inner lateral surface 944 and the media drop to enhancestability of the media drop. As a result, fluid flow resulting frommotion of the media drop can be reduced, which can reduce the likelihoodof embryos or pluripotent cells being pulled out of the micro-wells 910due to motion of the media drop.

FIG. 9E illustrates a cross-section view of the micro-well 910,according to an embodiment of the invention. In one embodiment, a lowersurface 960 of the micro-well 910 may be curved. For example, a firstdepth 962 at a center 968 of the micro-well 910 may be in the range fromabout 1.1 to about 1.5 times as large as a second depth 964 at a lateralperiphery 966 of the micro-well 910, such as about 1.2 times, about 1.3times, or about 1.4 times. Alternatively, the lower surface 960 of themicro-well 910 may be substantially planar, such that the first depth962 is substantially equal to the second depth 964.

FIG. 9F illustrates a cross-section view of a micro-well 910, accordingto an embodiment of the invention. The micro-well 910 is in manyrespects similar to the micro-well 910 described with reference to FIGS.9A, 9B, and 9E, so differences are described here. A lower surface 970of the micro-well 910 may be conical. For example, the lower surface 970may slope downwardly, and substantially linearly, from the lateralperiphery 966 to the center 968 of the micro-well 910. As described withreference to FIG. 9E, the first depth 962 may be in the range from about1.1 to about 1.5 times as large as the second depth 964, such as about1.2 times, about 1.3 times, or about 1.4 times.

FIG. 10 illustrates a system 1000 for automated imaging of humanembryos, oocytes, or pluripotent cells including an apparatus 1002 forautomated dish detection and well occupancy determination, according toan embodiment of the invention. The automated detection of themulti-well culture dish and the determination of well occupancy areprocessing performed prior to the automated imaging of human embryos.For the subsequent description with reference to FIGS. 10 to 25, themulti-well culture dish is referred to as the multi-well culture dish900 as described with reference to FIG. 9A, though it is contemplatedthat the multi-well culture dish can also correspond to the multi-wellculture dish 930 as described with reference to FIG. 9C, or to anysimilar multi-well dish where detection of the dish and determination ofoccupancy of wells included in the dish can be performed in a similarmanner.

The system 1000 includes a microscope controller 1001, which maycommunicate via a transmission channel 1004 with a set of microscopeswith imaging cameras 1010A-1010N. The microscope controller 1001 may beconnected to each microscope with imaging camera 1010 via apoint-to-point connection, or may be connected to multiple microscopeswith imaging cameras 1010 via a network. In one embodiment, themicroscope controller 1001 includes standard components, such asconnection interfaces 1014, a CPU 1016, and an input/output module 1018,which communicate over a bus 1012. In one embodiment, a memory 1006connected to the bus 1012 stores a set of executable programs that areused to implement the apparatus 1002 for automated detection of amulti-well culture dish and determination of occupancy of a plurality ofmicro-wells included in the multi-well culture dish. Alternatively, aprocessing device (such as circuitry, not shown) connected to the bus1012 can be used to implement the apparatus 1002 for automated detectionof a multi-well culture dish and determination of occupancy of aplurality of micro-wells included in the multi-well culture dish. Themicroscope controller 1001 may be connected to a server 1009 via atransmission channel 1011, which may be a point-to-point connection or anetwork. The server 1009 may include a dashboard for providing statusinformation and parameters determined based on analysis of images of ahuman embryo or pluripotent cell generated by the microscopes withimaging camera 1010.

In an embodiment of the invention, the memory 1006 stores executableinstructions establishing a dish detection module 1020, a well locationdetermination module 1022, a well occupancy determination module 1024,and a display module 1026. Alternatively, the processing device (notshown) includes the dish detection module 1020, the well locationdetermination module 1022, the well occupancy determination module 1024,and the display module 1026.

FIG. 11 is a series 1100 of images depicting the development of anembryo, and specifically of a human embryo. The series 1100 of imagesincludes eight images labeled (a), (b), (c), (d), (e), (f), (g), and(h), from left to right respectively. Image (a) identifies an embryo atthe time of fertilization. The embryo includes the zona pellucida 1102,a cell 1104, also referred to herein as a blastomere, and the polar body1106. Image (b) identifies an embryo at the time of the first cleavageevent, image (c) identifies the embryo after the second cleavage event,and image (d) identifies the embryo after the third cleavage event andbefore compaction, whereas image (e) identifies the embryo after thethird cleavage event and after compaction. Images (d) and (e) canrepresent an embryo at approximately 3 days of development.

As seen in image (f), the embryo begins cavitation and an inner cellmass 1108, a trophectoderm 1112, and a cavity 1110 are formed. Theembryo and the cavity 1110 grow as depicted in image (g), which imagedepicts an embryo at approximately day 5 of development. Atapproximately day 6 of development, the embryo hatches out of the zonapellucida 1102 as indicated in image (h).

FIG. 12 is a flowchart illustrating one embodiment of a process 1200 forpredicting viability of an embryo. This viability prediction caninclude, for example, a prediction of a ploidy status of the embryo suchas euploidy or aneuploidy, the prediction of the likelihood of theembryo implanting, or the like. In some embodiments, this viabilityprediction can include a recommendation for selection of the embryo, arecommendation for de-selection of the embryo, or a non-recommendationof the embryo. In some embodiments, a recommendation for selection caninclude a recommendation to use one or several embryos, which use caninclude the implantation of one or several embryos, and in someembodiments a recommendation for de-selection can include arecommendation to not use or to dispose of one or several embryos.

In some embodiments, this viability prediction can be provided to theuser via, for example, the dashboard that can be accessible to the uservia the input/output module 1018. In some embodiments, the viabilityprediction can be presented in the form of a ranking of one or severalembryos, for example from most likely to be viable and/or most likely tobe euploid to least likely to be viable and/or least likely to beeuploid. In some embodiments, the viability prediction can includesorting some or all of the embryos amongst one or several categories,which one or several categories can each be, for example, associatedwith a likelihood of viability and/or a likelihood of euploidy. In oneembodiment, for example, the embryos can be sorted amongst two, three,four, five, or six categories with the first category associated withthe greatest likelihood of viability and/or euploidy, with the fifthcategory associated with the smallest likelihood of viability and/oreuploidy, and the remaining categories having likelihoods of viabilityand/or euploidy intermediate between the first and fifth categories.

The process 1200 begins at 1202 wherein a series of images, andspecifically wherein a series of time lapse images 1201 is received. Insome embodiments, this series of images 1201 can be of a culture dish,such as multi-well culture dish 215. In some embodiments, the culturedish can be located on the stage of the microscope. In some embodiments,this series of images 1201 can include one or several embryos such ashuman embryo located in, for example, one or several of the plurality ofwells of the multi-well culture dish 215 can be received from one orseveral of the microscopes with imaging cameras 1010 depicted in FIG.10. In some embodiments, this series of images 1201 may be generatedand/or captured by one or several cameras of the one or several of themicroscopes with imaging cameras 1010.

Referring to the series of time-lapse images 1201, the series 1201 caninclude any desired number of images that can, for example have aconstant time interval between images or a varied time interval betweenimages. These images can comprise image data that can, for example,include data identifying an attribute of some or all of the pixelsforming each of the images including, for example, a pixel color,brightness, intensity, contrast, or the like.

In some embodiments, this series 1201 can extend from a first image t(1)to a final image t(n). In some embodiments, the first image t(1) can bethe first image captured by the camera, and in some embodiments, thefirst image t(1) can be designated based on its position relative to ananchor. This anchor can be, for example, a common and/or universalembryonic event such as, for example, one of the first, second, andthird cleavage events, cavitation, hatching, or the like. In someembodiments, this position relative to the anchor can be a number ofimage frames before or after the anchor, an amount of time before orafter the anchor, or the like. In one embodiment, for example, theanchor can be the first cleavage event, and the position relative to theanchor can be, for example, at a frame between 0 and 5,000 frames beforeor after the anchor event, at a frame between 0 and 1,000 frames beforeor after the anchor event, at a frame between 0 and 500 frames before ofthe anchor event, at a frame between 0 and 200 frames before or afterthe anchor event, at a frame between 0 and 100 frames before or afterthe anchor event, at a frame between 25 and 75 frames before or afterthe anchor event, or any other or intermediate position relative to theanchor. In some embodiment, the anchor can enable the use of a wavelettransform.

After the series of images has been received, the process 1200 proceedsto block 1204 wherein a mask is generated. In some embodiments, the maskcan be placed over portions of one or several of the series of images toblock those portions of the one or several of the series of images tofacilitate viewing and/or analysis of the unmasked portions of the oneor several of the series of images. Thus, in some embodiments, the mask,also referred to herein as an embryo mask, distinguishes between a firstportion of the image and a second portion of the image. In someembodiments, the first portion contains the image of the embryo, and thesecond portion does not contain the image of the embryo.

In some embodiments a mask can be generated for each of the images inthe series of images and can thus be unique for the image for which itis generated, and in some embodiments, a mask can be generated for aplurality of the images in the series of images. As used herein,“background” refers to portions of one or several imagesblocked/obscured by the mask, and is used herein, “foreground” refusedto portions of one or several images that are not blocked/not obscuredby the mask.

One embodiment of an exemplary process 1300 for the creation of the maskis shown in FIG. 13. This process 1300 is depicted by a series of imagesof a mask created for a hatching embryo. These images progress from leftto right, starting with image (a) and ending with image (c). In theseimages, the zona pellucida and the area enclosed by the zona pellucidaforms the left-most, white, spherical portion and the blastocycst asdefined by the trophectoderm forms the right-most, white portions of theimage. In images (a) and (b), the cavity within the blastocyst isvisible as the black enclosed area within the right-most, white portionof the image. This cavity portion is filled, and thus appears white inimage (c). The use of masks as shown in FIG. 13 can enable evaluation ofimage aspects, and extraction of one or several image features relatingto the cavity such as the area of the cavity, the perimeter of thecavity, the area of the embryo including or excluding the cavity area,or the like. Further, the use of masks as shown in FIG. 13 can enableevaluation of image aspects and extraction of one or several imagefeatures relating to the hatching, the zona pellucida, thetrophectoderm, the inner cell mass, and/or other embryonic structures.

This process 1300 begins with the evaluation of portions of one orseveral images of the series of images for mask classification. In someembodiments, these portions can be, for example, one or several pixels,one or several groups of pixels, or the like. This mask classificationcan include, for example, determining whether the evaluated portions arepart of the background or are part of the foreground. In someembodiments, evaluated portions are identified as part of the backgroundif they are outside of the image of the embryo and thus not part of theimage of the embryo, and evaluated portions are identified as part ofthe foreground if they are inside of the embryo and thus part of theembryo. This evaluation can be performed by a mask classifier which canbe, for example, a software module located within the server 1009 orwithin the microscope controller 1001.

After the evaluation of portions of one or several images of the seriesof images for mask classification, the mask classifier can generate andoutput a preliminary mask, also referred to herein as a preliminary maskimage. In some embodiments, the mask classifier can generate and outputa preliminary mask based on the results of the evaluation of portions ofone or several images of the series of images for mask classification,and specifically based on results of the evaluation of portions of oneor several of the images of the series of images as inside of the maskarea or outsides of the mask area. An exemplary embodiment of apreliminary mask is depicted in image (a) of FIG. 13. As seen in image(a), the preliminary mask can include one or several holes (see as blackportions within the left-most, white, circular portion of the image (a).

After the preliminary mask is been generated, the process 1300 proceedsto refine the preliminary mask. In some embodiments, the refinement caninclude identifying one or several continuous, foreground components.These components can be identified via Connected Component Analysis, andin some embodiments, all but one or several designated identifiedcomponents can be removed from the foreground, and particularly, all butthe largest one or several of these components can be removed from theforeground. In some embodiments, Connected Component Analysis can beperformed using any commercially or open source available ConnectedComponent Analysis software or code.

The refinement can further include the identification and filing of oneor several holes enclosed, or partially enclosed within the foregroundportions of the mask. This can be performed via hole filing, dilation,or other post processing technique. In some embodiments, the hole filingand/or dilation can be performed using available hole filing and/ordilation software or code.

Through the Connected Component Analysis and the hole filing, the maskcan be refined such that the foreground and background portions of themask are more continuous as depicted in images (b) and (c). In someembodiments, this refinement can be an iterative process involvingrepeated steps. In the images of FIG. 13, image (b) has been iterativelyless-refined than image (c). Thus, as seen in image (b), the cavity isstill unfilled in the right-most, white portion of the image.Accordingly, the cavity of the embryo is in the background. As furtherseen in image (b), the white image portions surrounding the cavity aremore continuous than in image (a) and the holes have been removed fromleft-most, white portion of the image (b).

In some embodiments, a first mask consistent with image (c) can becreated. Further, in some embodiments, a series of first masks can becreated for some or all of the images in the series of images, andparticularly, a unique first mask can be created for each of the imagesin the series of images from the image in the series of images for whichthat unique first mask was created. These first masks can be stored in,for example, the memory 1006.

This first mask can place a first area in the background. Thus, thisfirst mask can obscure the first area when overlaying one image of theseries of images 1201. In some embodiments, this second area can includesome or all of the areas outside of the embryo, including outside of thezona pellucida.

In some embodiments, an area of the first mask can be determined. Asused herein, the area of the first mask can refer to the area of theforeground portion unobscured by the first mask when the first mask isoverlaid on one of the images in the series of images, and particularlywhen the first mask is overlaid on the image in the series of imagesfrom which the first mask was created. In some embodiments, this firstarea of the mask can be determined by determining the number of pixelsin the foreground when the first mask is overlaid on the image in theseries of images. This first mask area can be determined for some or allof the images in the series of images using the first mask created foreach of the some or all of the images in the series of images. Thesefirst mask areas can be stored in the memory 1006.

In some embodiments, a second mask consistent with the image (b) can becreated. Further, in some embodiments, a series of second masks can becreated for some or all of the images in the series of images, andparticularly, a unique second mask can be created for each of the imagesin the series of images from the image in the series of images for whichthat unique second mask was created. These second masks can be storedin, for example, the memory 1006.

This second mask places a second area in the background. Thus, thesecond mask obscures the second area when overlaying one image of theseries of images 1201. This second area can include the first area, aswell as one or several internal portions of the embryo, and specificallyone or several portions of the cavity. Thus, as seen in image (b), thearea outside of the embryo including outside of the zona pellucida, andthe area of the cavity are in the background. A mask that has gonethrough refinement is referred to herein as one of the first mask or thesecond mask, or as a refined mask or a final mask.

In some embodiments, an area of the second mask can be determined. Asused herein, the area of the second mask can refer to the area of theforeground portion unobscured by the second mask when the second mask isoverlaid on one of the images in the series of images, and particularlywhen the second mask is overlaid on the image in the series of imagesfrom which the second mask was created. In some embodiments, this secondarea of the mask can be determined by determining the number of pixelsin the foreground when the second mask is overlaid on the image in theseries of images. This second mask area can be determined for some orall of the images in the series of images using the second mask createdfor each of the some or all of the images in the series of images. Thesesecond mask areas can be stored in the memory 1006. Further, in someembodiments, one or several calculations can be performed using thefirst and second mask areas. In some embodiments, this can includedetermining differences for one or several of the first and second masksarea.

Returning again to FIG. 12, after the mask has been generated andapplied, superimposed, and/or overlaid on the image with which it isassociated, the process 1200 proceeds to block 1206, wherein one orseveral image features, also referred to herein as image-based features,are extracted from one or several of the images of the series of images.In some embodiments, the applying, superimposing, or overlaying of themask on the image with which the mask is associated can enableidentification of at least one of: cavitation; and hatching.Specifically, in some embodiments, the use of the masks described hereinenable detection of a feature of one or several of the images based onthe embryo mask.

In some embodiments, these one or several features can be automaticallyextracted, and in some embodiments, these one or several features can beonly automatically extracted such that they are not manually measuredand/or determined. Thus, in such embodiments, these one or several imagefeatures can be extracted from the series of images by, for example, theserver 1009 or the microscope controller 1001 without any substantivehuman inputs or any substantive human action. These one or several imagefeatures can be stored in, for example, the memory 1006.

The image features can be any features that can be extracted from one orseveral of the images of the series of images. In some embodiments theseimages features relate to one or several of the images of the series ofimages, and in some embodiments, these image features relate to all ofthe images of the series of images. These image features can, forexample, relate to one or more of: cavitation; hatching; embryoexpansion; and embryo collapse, and in some embodiments, these imagefeatures can relate to at least one of: an area of the embryo; an areaof a cavity of the embryo; a perimeter of the embryo; and a convex hullof the embryo. In some embodiments, image features relating tocavitation, also referred to herein as cavitation features, can relateto at least one of: embryo image area; cavity image area; a change inembryo image area over time; a change in cavity image area over time;embryo image perimeter; and convex hull. In some embodiments, thecavitation features can relate to: an average cepstrum of an embryoattribute; a final embryo area; a maximum embryo area; an average embryoarea; a number of cavitation peaks; a final cavitation area; a maximumcavitation area; a maximum embryo area difference; a maximum cavitationarea difference; a mean ration of cavitation and embryo areas; a meanarea of cavitation; and a maximum ratio of cavitation and embryo areas.

The image features can be extracted in a variety of different ways thatdepend on the details of the specific image feature being extracted. Insome embodiments, for example, the image features can be extracted basedon one or several properties of one or both of the first and secondmasks. Thus, in some embodiments, one or image features can bedetermined based on: the area and/or perimeter of one or both of themasks; the difference in areas of the first and second masks; and theshape of all or portions of one or both of the first and second masks.

In some embodiments, the extraction of the image features can furtherinclude the application of one or several transforms or analysistechniques to data extracted from all or portions of the series ofimages. These transforms or analysis techniques can include, forexample, at least one of: a Fourier transform, including a DiscreteFourier Transform (DFT); a cepstrum transform that can be calculated asthe inverse DFT of the log magnitude of the DFT of the extracted data;and a wavelet transform such as a discrete wavelet transform. In someembodiments, the discrete wavelet transform comprises one of: Haarwavelet; Daubechies wavelets; Symlets wavelets; Battle-Lemarie wavelets;and Biorthogonal wavelets.

To calculate a cepstrum, a time-series of extracted data is createdacross some or all of the images of the series of images. In someembodiments, this extracted data can be an untransformed image featureand the time series of extracted data can be a 1D vector offloating-point numbers. The following computation can be applied to this1D vector:|

⁻¹{log(|

{f(t)}|²)}|²In this expression,

represents the Fourier Transform and

⁻¹ represents the Inverse Fourier Transform. After applyingdiscretization on the continuous variable t f (t)→f_(t) (where t∈Z and Zindicates the set of integer values), these transforms become (for Ntime-samples):

${{\hat{f}}_{\omega} = {{\left\{ f_{t} \right\}} = {\sum\limits_{t = 0}^{N - 1}{f_{t}e^{{- 2}\;\pi\; i\; w\;{t/N}}}}}},{\omega \in Z}$

and:

f t = - 1 ⁢ { f ^ ω } = ∑ ω = 0 N - 1 ⁢ f ^ ω ⁢ e + 2 ⁢ ⁢ π ⁢ ⁢ i ⁢ ⁢ w ⁢ ⁢ t / N ,t ∈ ZThe output of these expressions is a transformed 1D vector correspondingto the 1D vector to which the transform was applied. In someembodiments, the mean value of that transformed 1D vector can becalculated, which mean value is referred to herein as a “mean cepstrum.”

Exemplary image features are listed in Table 1, below. Any one of theimage features listed in Table 1 may be used alone or in combinationwith each other or other features.

TABLE 1 List of Image Features Image Feature Description/Referencedescribing Image Feature Mean This is the average cepstrum of the embryoarea for some or all of the images of Cepstrum of the series of images.The area of the embryo including the cavity is determined the Embryo bythe area of the first mask. The mean cepstrum of the embryo area is Areacalculated by: determining the area of the embryo in some or all of theimages of the series of images; calculating the cepstrum for each of thedetermined areas of the embryo; and calculating the average cepstrum.Mean This is the average cepstrum of embryo perimeter for some or all ofthe images Cepstrum of of the series of images. The perimeter of theembryo is the geodesic distance the Embryo around the mask. The meancepstrum of the embryo perimeter is calculated by: Perimeter determiningthe perimeter of the embryo in some or all of the images of the seriesof images; calculating the cepstrum for each of the determinedperimeters of the embryo; and calculating the average cepstrum. MeanThis is the average cepstrum of the number of convex hull points forsome or all Cepstrum of of the images of the series of images. Thenumber of convex hull points is the the Number of minimum number ofpoints located on the perimeter of the mask or on the Convex Hullperimeter of the image of the embryo that define lines that togetherPoints circumscribe the mask or the image of the embryo. FIG. 14 depictsa convex hull 1400 generated so as to circumscribe an image of embryo1402. As seen in FIG. 14, the convex hull 1400 is generated by aplurality of line segments 1404 extending between points 1406 which arelocated on the perimeter of the embryo 1400 and/or mask. In someembodiments, the points 1406 can be positioned on the perimeter of theimage of the embryo 1402 and/or mask such that the minimum number ofpoints 1406 is generated to define the smallest number of line segments1404 to circumscribe and enclose the embryo 1402 and/or mask without theline segments crossing into or over the image of the embryo 1402 and/ormask. The mean cepstrum of the number of convex hull points iscalculated by: determining the number of convex hull points in some orall of the images of the series of images; calculating the cepstrum foreach of the determined numbers of convex hull points; and calculatingthe average cepstrum. Final Embryo This is the area of the embryoincluding the cavity at the final image in the Area series of images.This final area is the area of the first mask for that final image. MaxEmbryo This is the maximum area of the embryo, including the cavity, inany image of Area the series of images. This maximum area can becalculated by determining the area of the first mask for each of some orall of the images of the series of images, and identifying the maximumvalue of those determined areas. Mean Embryo This is the average area ofthe embryo including the cavity calculated for some Area or all of theimages of the series of images. This average embryo area is calculatedby determining the area of the first mask for each of the some or all ofthe images of the series of images, and calculating the average of thosedetermined areas. Number of This is the number of maxima of cavitationvolume in the series of images. This Cavitation is calculated bydetermining the cavitation area in some or all of the images in Peaksthe series of images, identifying images with a larger cavitation areathan their adjacent images, and incrementing a count for each suchidentified image. Final This is the area of the cavity at the finalimage in the series of images. This final Cavitation area is the area ofthe first mask for that final image minus the second mask for Area thatfinal image. Max This is the maximum area of the cavity, in any image ofthe series of images. Cavitation This maximum area can be calculated bydetermining the area of the cavity by Area subtracting the area of thesecond mask from the area of the first mask for each of some or all ofthe images of the series of images, and identifying the maximum value ofthose determined areas. Max Embryo This is the maximum absolutedifference in embryo area between adjacent Area images in the series ofimages. This is calculated by: determining the area of the Differencefirst mask in some or all of the images of the series of images;identifying one or several pairs of adjacent images in the series ofimages; taking the absolute value of the differences between the areasof the images of the identified pairs of adjacent images; andidentifying the largest of the absolute values of these differences. MaxThis is the maximum difference in cavitation area between adjacentimages in Cavitation the series of images. This is calculated by:determining the difference between Area the first and second masks(cavitation area) for some or all of the images in the Difference seriesof images; identifying one or several pair of adjacent images in theseries of images; and identifying the largest of the differences betweenthe cavitation area of the adjacent images in the pairs. Mean Ratio ofThis is mean ratio of cavitation and embryo areas for some or all of theimages Cavitation and in the series of images. This is calculated bycalculating the ratio of cavitation Embryo Areas and embryo areas forsome or all of the images in the series of images by dividing thecavitation area (area of the first mask minus the area of the secondmask) by the embryo area (area of the first mask). The average value ofthese ratios is then calculated. Mean This is the average cavitationarea for some or all of the images in the series of Cavitation images.This is calculated by identifying the cavitation area for some or all ofArea the images in the series of images (area of the first mask minusthe area of the second mask); and calculating the average value of thesecavitation areas. Maximum This is the maximum value of the ratio ofcavitation and embryo areas for some Ratio of or all of the images inthe series of images. This is calculated by calculating the Cavitationand ratio of cavitation and embryo areas for some or all of the imagesin the series Embryo of images by dividing the cavitation area (area ofthe first mask minus the area of the second mask) by the embryo area(area of the first mask); and identifying the largest calculated ratio.2^(nd) Max This is the second largest area of the embryo, including thecavity, in any image Embryo Area of the series of images. This secondlargest area can be calculated by determining the area of the first maskfor each of some or all of the images of the series of images, andidentifying the second largest value of those determined areas. 3^(rd)Max This is the third largest area of the embryo, including the cavity,in any image Embryo Area of the series of images. This third largestarea can be calculated by determining the area of the first mask foreach of some or all of the images of the series of images, andidentifying the third largest value of those determined areas. 2^(nd)Max This is the second largest area of the cavity, in any image of theseries of Cavitation images. This second largest area can be calculatedby determining the area of Area the cavity by subtracting the area ofthe second mask from the area of the first mask for each of some or allof the images of the series of images, and identifying the secondlargest value of those determined areas. 3^(rd) Max This is the thirdlargest area of the cavity, in any image of the series of images.Cavitation This third largest area can be calculated by determining thearea of the cavity by Area subtracting the area of the second mask fromthe area of the first mask for each of some or all of the images of theseries of images, and identifying the third largest value of thosedetermined areas. Mean This is the average cepstrum value of thestandard deviation of Hessian features Cepstrum of for some or all ofthe images in the series of images, which Hessian features can Std. Dev.Of be a, for example, the coarsest resolution. In some embodiments, theHessian is Hessian the result of: 1) coarse-grain smoothing such asoccurs, for example, when Features using a Gaussian filter with standarddeviation of, for example, 5 pixels; and (2) applying a 2nd orderderivative filtering in X and Y at some or all of the pixels in aselected image. These yield a Hessian matrix for the pixels to whichthis was applies. In some embodiments, the smallest eigenvalue can beextracted from this matrix, and the standard deviation of thesesmallest, extracted eigenvalues can be calculated for each image acrosssome or all of the pixels in that image. The cepstrum transform isapplied to the standard deviation of the smallest eigenvalues, and theaverage value of the result of this transform is calculated. Mean Thisis the average cepstrum value of the standard deviation of continuityfor Cepstrum of some or all of the images in the series of images. Thecontinuity is obtained Std. Dev. Of from the embryo boundary by:expressing the boundary in polar coordinates Continuity using the embryoregion's center of mass as the origin; and calculating the absolutevalue of changes in the radius as a function of change in angle forfixed angular increments. This calculated value is mathematicallyexpressed as ${\frac{\Delta\; r}{\Delta\;\theta}}.$ The absolute valueof the difference between calculated value$\left( {\frac{\Delta\; r}{\Delta\;\theta}} \right)$ and the averageof all of those calculated values$\left( {\frac{\Delta\; r}{\Delta\;\theta}} \right)$ is calculated,and the standard deviation of those absolute values is calculatedresulting in the creation of a time series of standard deviations. Thecepstrum transform is applied to this time series of standarddeviations, and the output of the cepstrum transform is averaged. MeanThis is the average cepstrum value of the mean continuity for some orall of the Cepstrum of images in the series of images. The continuity isobtained from the embryo Mean boundary by: expressing the boundary inpolar coordinates using the embryo Continuity region's center of mass asthe origin; and calculating the absolute value of changes in the radiusas a function of change in angle for fixed angular increments. Thiscalculated value is mathematically expressed as${\frac{\Delta\; r}{\Delta\;\theta}}.$ The absolute value of thedifference between each calculated value$\left( {\frac{\Delta\; r}{\Delta\;\theta}} \right)$ and the averageof all of those calculated values$\left( {\frac{\Delta\; r}{\Delta\;\theta}} \right)$ is calculated,and the mean of those absolute values is calculated resulting in thecreation of a first time series of means. The cepstrum transform isapplied to this time series of means and the output of the cepstrumtransform is then averaged.

Although the above identifies the first, second, and third maximumvalues for embryo area and cavitation area, additional maximum valuesfor embryo area and cavitation area can be used. These can include, forexample, the 4^(th) largest, 5^(th) largest, 6^(th) largest, 7^(th)largest, 8^(th) largest, 9^(th) largest, 10^(th) largest, and/or anyother maximum values.

In some embodiments, these one or several image features were identifiedand/or selected based on their effectiveness in contributing to thegeneration of a viability prediction. For this selection, a set of datawas received randomly partitioned into a training set and a testing set.In some embodiments, this partition was done following a fixedproportion, also known as a split-ratio. After the partitioning of thedata, a statistical model was trained with the training-set and theeffectiveness of the model was determined with the testing set. Thesesteps were iterated until a desired number of iterations had occurred,and then the average predictive power of the model is calculated.

In some embodiments, the predictive power of the statistical model, andthe effectiveness of image features or other features used in thatstatistical model can be determined through the calculation of a Fisherscore, which is a measure of inter-class scatter divided by intra-classscatter. The Fisher score is given by:

$S = {\frac{\sigma_{between}^{2}}{\sigma_{within}^{2}} = {\frac{\left( {{w^{T}\mu_{+ 1}} - {w^{T}\mu_{- 1}}} \right)^{2}}{{w^{T}\Sigma_{+ 1}w} + {w^{T}\Sigma_{- 1}w}} = \frac{\left( {w^{T}\left( {\mu_{+ 1} - \mu_{- 1}} \right)} \right)^{2}}{{w^{T}\left( {\Sigma_{+ 1} + \Sigma_{- 1}} \right)}w}}}$

andw∝(Σ₊₁+Σ⁻¹)⁻¹(μ₊₁−μ⁻¹)In the expressions above, the subscripts indicate 2 labels indicative ofdesired predicted states or classifications. In this specific instance,the classifications are viable/non-viable, which can also includeeuploid/non-euploid, euploid/aneuploid, aneuploidy/non-aneuploid,implant/non-implant, or the like. As specifically identified, “+1” inthe above expressions identifies euploid and “−1” identifies aneuploidy.In the above expressions, μ₊₁ indicates the mean value offeature-vectors which are labeled as +1 (a vector), μ⁻¹ indicates themean value of feature-vectors which are labeled as 11 (a vector), Σ₊₁indicates the covariance (a matrix, defined as the mean-centered 2^(nd)moment) of the feature-vectors which are labeled as +1, and Σ⁻¹indicates the covariance (a matrix, defined as the mean-centered 2^(nd)moment) of the feature-vectors which are labeled as −1.

After Fisher scores are calculated, the effectiveness of the modeland/or of one or several of the image features or other parameters canbe ascertained with the Fisher score. In some embodiments, for example,a larger Fisher score can be indicative of a more effective model and/orfeature, and thus features can be selected that have the highest Fisherscores and/or a model can be accepted when it has the highest Fisherscore of a group of models, or when its Fisher score reaches apredetermined threshold level.

In some embodiments the image features extracted from all or portions ofthe series of images can be supplemented by one or several otherparameters. These one or several other parameters can be inputted intothe system and can, for example, relate to the age of the human sourceof the egg at the time of egg harvesting, the age of the egg sinceharvesting, past medical history of the human source of the egg,information relating to past success or failure with IVF, or the like.In some embodiments, these one or several other parameters can bereceived from a user via the input/output module 1018 and can be storedin the memory 1006.

In some embodiments, these other parameters can include, for example,one or several prior recommendations or prior viability predictions. Insome embodiments, for example, images can be received for evaluationshortly after they have been captured. In such an embodiment, it can beadvantageous to generate evaluate viability based on the images alreadyreceived and not wait for the completion of capturing all of the imagesin the series of images. In some such embodiments, an initialrecommendation can be generated based on the evaluated one or severalimages, but this recommendation can be updated as further images arereceived.

In some embodiments, different parameters may be used in generating arecommendation for images of embryos in different developmental stages.In one embodiment, for example, a first recommendation can be generatedbased on images generated up to the third day of development, ascorresponding with images (d) and (e) of FIG. 11. This firstrecommendation can include a viability prediction identifying thelikelihood of the embryo associated with the prediction forming ablastocyst. In such embodiments, the first recommendation can begenerated based on image features extracted from the received imagesand/or from one or several other parameters. In one embodiment, forexample, the first recommendation can be generated based on: theduration of the first cytokinesis, the length of the time intervalbetween the first cytokinesis and the second cytokinesis, the length ofthe time interval between the second cytokinesis and the thirdcytokinesis, the age of the human source of the egg at the time of eggharvesting, a cell count, which can be a manual cell count, at the endof the time period for the first recommendation, and a result of amorphological analysis, which can be manually performed. In someembodiments, the age of the human source of the egg at the time of eggharvesting, a manually performed cell count, and a manually performedmorphological analysis are all examples of manual biologicalmeasurements.

In some embodiments, the length of the time interval between the firstcytokinesis and the second cytokinesis can be, for example, the timefrom 2-cell embryo to 3-cell embryo, the time from the end of the firstcytokinesis to the end of the second cytokinesis, the duration of timeas a 2 cell embryo, and/or any other measure of the duration of thistime period. In some embodiments, the length of the time intervalbetween the second cytokinesis and the third cytokinesis can be, forexample, the time from 3-cell embryo to 4-cell embryo, the time from theend of the second cytokinesis to the end of the third cytokinesis, theduration of time in which the embryo was 3 cell embryo, and/or any othermeasure of this time period.

After the one or several image features have been extracted, the process1200 proceeds to decision block 1208, wherein it is determined if therewere any prior recommendations. In one embodiment, for example, a priorrecommendation can be generated based on images from t(0) to t(x), and afurther recommendation can be based on, for example, images from t(0) tot(n), images from t(x) to t(n), and/or any other set of images. If it isdetermined that there is a prior recommendation, then the process 1200proceeds to block 1210, wherein the prior recommendation is retrievedfrom, for example, the memory 1006.

After the prior recommendation has been retrieved, or returning again todecision block 1208, if it is determined that there are no priorrecommendations, the process 1200 proceeds to block 1212, wherein theone or several image features are inputted into a machine learningmodule, also referred to as a classification module, configured forgeneration of a viability prediction based on the received imagefeatures, and specifically based on one or several features such as acavitation or cavitation related feature, a feature including a cepstrumsuch as a cavitation cepstrum, or any other image feature identifiedabove. Thus, in some embodiments, the machine learning module cangenerate a viability predication, with can include a prediction ofeuploidy or aneuploidy, a prediction of the likelihood of the embryoimplanting, or the like.

In some embodiments, and as depicted in FIG. 12, this machine learningmodule can be one or several classifiers that are usable for outcomedetermination. Any suitable classifier may be employed. In someembodiments, the classifier is based on a machine learning algorithm.The classifier may be an AdaBoost (adaptive boosting) classifier, aSupport Vector Machine (SVM), a Naïve Bayes classifier, a classifierusing an ensemble method such as a Random Forest classifier, or aBoosting Tree. In some embodiments, an ensemble method classifier thatuses multiple learning algorithms to obtain better predictiveperformance than could be obtained by using any of that multitude oflearning algorithms alone. Thus, this in effect groups several “weaklearners” together to form a “strong learner.”

In some embodiments, classifiers can be arranged into multiple levels,and in some embodiments, each of these multiple levels can include oneor several classifiers. In some embodiments, a refining algorithm can beapplied to the output of the last image classifier to further refine theclassification of the image. In some embodiments, the refining algorithmrefines the classification of each image based on a temporal imagesimilarity measure of the image. In some embodiments, the refiningalgorithm is a dynamic programming algorithm for finding the most likelyclassification of the images included in the time-lapse series ofimages. In some embodiments, the refining algorithm is a Viterbialgorithm.

After the one or several image features are inputted into a machinelearning module, the classifier generates a viability prediction. Insome embodiments, this viability prediction can be based only on theinputted image features, and in some embodiments, this viabilityprediction can be based on the inputted image features and one orseveral other parameters such as, for example, age of the human sourceof the egg at the time of egg harvesting, a prior recommendation, or thelike. In some embodiments, this viability prediction can be based on acavitation or cavitation related feature, a feature including a cepstrumsuch as a cavitation cepstrum, or any other image feature identifiedabove.

In some embodiments, this viability prediction can be automaticallygenerated, and specifically can be generated based on the inputtedfeatures by, for example, the server 1009 or the microscope controller1001 without any manual biological measurements, without any substantivehuman inputs, and/or without any substantive human action. As usedherein a “substantive human input” and a “substantive human action” donot include inputs or actions to start or continue all or portions ofthe process 1200 shown in FIG. 12 and/or relating to non-outcomedeterminative action or input. In some embodiments, the viabilityprediction can be stored in the memory 1006.

In some embodiments, the classifier can generate the viabilityprediction based on training of the classifier. This training can bebased on a data set, and particularly on a data set repeatedly, randomlypartition into a training set and a testing set. This training canproceed as discussed above until a desired measure of effectiveness ofthe classifier is achieved. As a result of the training of theclassifier, the classifier can generate a viability prediction based oninputs received by the classifier.

After the image features are inputted into the classifier and after theclassifier generates a viability prediction, the process 1200 proceedsto decision block 1214, wherein it is determined if additional imageshave been generated. In some embodiments, this can include determiningwhether further images have been generated and/or received that were notincluded in the generation of the viability prediction by the classifierin block 1212. If it is determined that there are additional images,then the process 1200 returns to block 1202, and proceeds as outlinedabove.

If it is determined that there are no additional images, then theprocess 1200 proceeds to block 1216, wherein the viability prediction isprovided. In some embodiments, the viability prediction can be providedto the user via the input/output module 1018 and specifically via theGUI. In some embodiments, the outputted viability prediction can includethe outputting of one or several categories into which one or severalimaged embryos have been placed, the outputting of a ranking of one orseveral embryos relative to each other, which ranking can identify theembryos that are most likely to be viable or least likely to be viable,or the like.

FIGS. 15 and 16 show the results of classification using age and thefirst 14 image features in Table 1. Specifically, FIG. 15 is a bar graphindicating the euploid rate of embryos placed into each of fivecategories (e.g. Cat 1, Cat 2, Cat 3, Cat 4, and Cat 5). As seen, Cat 1has the highest euploid rate at 92%, and Cat 5 has the lowest euploidrate at 38%.

FIG. 16 is a bar graph indicating the distribution of embryos in thesample from which the bar graph of FIG. 15 was generated. As seen, Cat 1includes approximately 14 embryos, Cat 2 includes approximately 45embryos, Cat 3 includes approximate 72 embryos, Cat 4 includesapproximately 70 embryos, and Cat 5 includes approximately 9 embryos.

An embodiment of the invention relates to a computer storage productwith a computer-readable medium having computer code thereon forperforming various computer-implemented operations. The term“computer-readable medium” is used herein to include any medium that iscapable of storing or encoding a sequence of instructions or computercodes for performing the operations described herein. The media andcomputer code may be those specially designed and constructed for thepurposes of the invention, or they may be of the kind well known andavailable to those having skill in the computer software arts. Examplesof computer-readable media include, but are not limited to: magneticmedia such as hard disks, floppy disks, and magnetic tape; optical mediasuch as CD-ROMs and holographic devices; magneto-optical media such asfloptical disks; and hardware devices that are specially configured tostore and execute program code, such as application-specific integratedcircuits (“ASICs”), programmable logic devices (“PLDs”), and ROM and RAMdevices. Examples of computer code include machine code, such asproduced by a compiler, and files containing higher-level code that areexecuted by a computer using an interpreter or a compiler. For example,an embodiment of the invention may be implemented using Java, C++, orother object-oriented programming language and development tools.Additional examples of computer code include encrypted code andcompressed code. Moreover, an embodiment of the invention may bedownloaded as a computer program product, which may be transferred froma remote computer (e.g., a server computer) to a requesting computer(e.g., a client computer or a different server computer) via atransmission channel. Another embodiment of the invention may beimplemented in hardwired circuitry in place of, or in combination with,machine-executable software instructions.

An embodiment of the invention can be implemented in hardware, such as afield programmable gate array (FPGA) or ASIC. The FPGA/ASIC may beconfigured by and may provide output to input/output devices.

The preceding merely illustrates the principles of the invention. It isappreciated that those skilled in the art may be able to devise variousarrangements which, although not explicitly described or shown herein,embody the principles of the invention and are included within itsspirit and scope. The illustrations may not necessarily be drawn toscale, and manufacturing tolerances may result in departure from theartistic renditions herein. There may be other embodiments of thepresent invention which are not specifically illustrated. Thus, thespecification and the drawings are to be regarded as illustrative ratherthan restrictive. Additionally, the drawings illustrating theembodiments of the present invention may focus on certain majorcharacteristic features for clarity. Furthermore, all examples andconditional language recited herein are principally intended to aid thereader in understanding the principles of the invention and the conceptscontributed by the inventors to furthering the art, and are to beconstrued as being without limitation to such specifically recitedexamples and conditions. Moreover, all statements herein recitingprinciples, aspects, and embodiments of the invention as well asspecific examples thereof, are intended to encompass both structural andfunctional equivalents thereof. Additionally, it is intended that suchequivalents include both currently known equivalents and equivalentsdeveloped in the future, i.e., any elements developed that perform thesame function, regardless of structure. The scope of the presentinvention, therefore, is not intended to be limited to the exemplaryembodiments shown and described herein. Rather, the scope and spirit ofthe present invention is embodied by the appended claims. In addition,while the methods disclosed herein have been described with reference toparticular operations performed in a particular order, it will beunderstood that these operations may be combined, sub-divided, orre-ordered to form an equivalent method without departing from theteachings of the invention. Accordingly, unless specifically indicatedherein, the order and grouping of the operations are not limitations ofthe invention. All references cited herein are incorporated by referencein their entireties.

What is claimed is:
 1. A method for implantation of human embryosselected with an imaging system, the method comprising: receivingtime-lapse images of at least one human embryo contained in a multiwellculture dish comprising a plurality of micro-wells; automaticallygenerating image-based features from the time-lapse images of the humanembryo; masking said image-based features to distinguish a firstimage-based feature portion from a second image-based feature portion;inputting the first image-based feature portion into a mask classifier;and automatically and directly generating a viability prediction withthe mask classifier from the first image-based feature portion;generating a recommendation for implantation from the viabilityprediction, wherein the implantation recommendation is selected from thegroup consisting of; i) selecting a euploid human embryo; and ii)deselecting an aneuploid human embryo; and implanting said selectedeuploid human embryo in a human.
 2. The method of claim 1, wherein theviability prediction comprises a prediction of euploidy or of aneuploidyin the human embryo.
 3. The method of claim 1, wherein the classifiercomprises one of: a Random Forest classifier; an AdaBoost classifier; aNaive Bayes classifier; Boosting Tree, and a Support Vector Machine. 4.The method of claim 1, wherein the image-based features relate to atleast one of: cavitation; hatching; embryo expansion; and embryocollapse.
 5. The method of claim 1, wherein the image-based featuresrelate to at least one of: an area of the embryo; an area of a cavity ofthe embryo; a perimeter of the embryo; and a convex hull.
 6. The methodof claim 1, wherein the image-based features relate to at least one of:average cepstrum of an embryo attribute; final embryo area; maximumembryo area; average embryo area; number of cavitation peaks; finalcavitation area; maximum cavitation area; maximum embryo areadifference; maximum cavitation area difference; mean ration ofcavitation and embryo areas; mean area of cavitation; and maximum ratioof cavitation and embryo areas.
 7. The method of claim 1, wherein theimage-based features relate to at least one of: a second highest embryoarea; a third highest embryo area; an average cepstrum of the number ofconvex hull points; a second highest cavitation area; a third highestcavitation area; an average cepstrum of traversal cost; an averagecepstrum of standard deviation of Hessian features; an average cepstrumof standard deviation of continuity; and an average cepstrum of meancontinuity.
 8. The method of claim 1, wherein the viability predictionis automatically and directly generated with the classifier from theimage-based features without any manual biological measurements.
 9. Themethod of claim 1, wherein the viability prediction is automatically anddirectly generated with the classifier only from the image-basedfeatures.
 10. The method of claim 1, wherein the viability prediction isautomatically and directly generated with the classifier from theimage-based features and from manual biological measurements.
 11. Anautomated imaging system for evaluation of human embryos to determine adevelopment potential, the system comprising: a computer to control thesystem, wherein said computer comprises a mask classifier software on acomputer readable medium; a stage configured to receive a multi-wellculture dish comprising a plurality of micro-wells containing a samplecomprising at least one human embryo; a time-lapse microscope incommunication with the computer, wherein the microscope is configuredto: acquire time-lapse images of the at least one embryo, wherein themask classifier generates a mask onto time-lapse images of the at leastone embryo: automatically generate an unobscured image-based featureportion and an obscured image-based feature portion from the time-lapseimages of the masked human embryo; input the unobscured image-basedfeature portion into the mask classifier software; automatically anddirectly generate a viability prediction with the mask classifiersoftware, wherein said viability prediction comprises a prediction ofeuploidy or of aneuploidy in the human embryo; and determine that the atleast one embryo has development potential for human implantation fromthe viability prediction.
 12. The system of claim 11, wherein directlygenerating the viability prediction comprises associating the humanembryo imaged in the received time-lapse images with one of a pluralityof ranked categories, wherein the categories are ranked according to alikelihood of viability.
 13. The system of claim 11, wherein thetime-lapse microscope is further configured to generating arecommendation from the viability prediction, and wherein a GraphicalUser Interface (GUI) on a display is configured to provide therecommendation with the display.
 14. The system of claim 13, wherein therecommendation comprises one of: a recommendation to select the humanembryo; and a recommendation to deselect the human embryo.
 15. Thesystem of claim 12, wherein the image-based features relate to at leastone of: cavitation; hatching; embryo expansion; and embryo collapse. 16.The system of claim 12, wherein the image-based features relate to atleast one of: an area of the embryo; an area of a cavity of the embryo;a perimeter of the embryo; and a convex hull.